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(54) Region-of-interest setting apparatus for respiration monitoring and a respiration 
monotoring system 



(57) A region*of-interest (ROI) setting apparatus for 
respiration monitoring and a respiration monitoring sys- 
tem are capable of setting an ROI in an automatic man- 
ner. Absolute values of subtractions between every 
successive pairs of frame images produced by a CCD 
camera 2 for a period of time equal to half a respiration 
period are calculated, then summed and stored. Based 
on the change information thus calculated and stored, 
the position and size of each changed region is calcu- 
lated. Provisional regions are set from the changed 
regions successively from the greatest to the smallest 



one thereof. In each of these provisional regions, 
processing is carried out to determine whether a grey 
level histogram indicative of distribution of the number of 
pixels for respective grey level values includes a twin 
peak characteristic having twin peaks each equal to or 
higher than a predetermined value, and whether the 
area of the corresponding changed region Is equal to or 
greater than a predetermined value. One of the provi- 
sional regions that satisfies these conditions is set as an 
ROI. 
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Description 

[0001] The present invention relates to a respiration 
monitoring system which is capable of continuously 
monitoring the respiratory state of a person, involving. 5 
for instance, visual sensing of a patient in a non-restric- 
tive manner. The invention also relates to apparatus for 
setting a region of interest for respiration monitoring. 
[0002] In recent years, the number of persons of 
advanced age has increased rapidly and it is desired to 
increase the number of staff who take care of bedridden 
elderly persons with attendant increasing personnel 
costs and medical care expenditure. This is an object of 
great public concern. 

[0003] Respiration monitoring systenr^ have been 
developed which are capable of continuously monitor- 
ing the respiratory state of a bedrkklen person through 
visual sensing in a non-restrictive manner. 
[0004] Fig. 1 3 shows a typical conventional respiration 
monitoring system as disclosed in "Automatic Measure- 
ments of the Number of Respirations under Conplete 
Non-restrictions by a Visual Sensing System" reported 
in the Proceedings of the 16tti Bio-Mechanism Society 
Conference held on November 25, 1995 by the Bio- 
Mechanism Society (pages 279 to 282). This respiration 
monitoring system includes a CCD camera 2 for imag- 
ing or photographing ttie chest of a patient 1 as an 
object to be imaged who is lying on a bed. and an image 
processor 3 connected to an output of tiie CCD camera 
2 for successively extiacting an inter-image difference 
or subtraction (i.e., subtraction between every two suc- 
cessive images) of continuous motion pictures at a con- 
stant rate. Predetermined image processing steps are 
performed so as to obtain or calculate a difference com- 
ponent thereby to measure the number of respirations 
per unit time. The system also includes a monitor 4 con- 
nected to an output of the image processor 3 for display- 
ing a visual indication of tiie detected data including the 
measured number of respirations per unit time, the state 
of respiration and/or for displaying an abnormal state in 
the case of an abnormality taking place. A setting input 
stage 5 allows an operator to input various settings such 
as a processing region in tiie form of a region of interest 
(hereinafter simply referred to as *'ROn of the image 
processor 3, a subtraction interval (a) in ttie image 
processing, etc. 

[0005] Fig. 14 is a flow chart showing operation (i.e.. 
respiration counting operation) of the above-mentioned 
conventional respiration monitoring system. Firstly in 
step S1, the image processor 3 successively receives 
image data from the CCD camera 2. and in step S2. it 
performs signed (+, -) suWraction of pixels between 
every adjacent two images that are spaced from each 
other by a predetermined frame interval (a). Subse- 
quentiy. in step S3, tiie image processor 3 performs sur- 
face integration within a ROI as follows. The extracted 
image data are subjected to signed sut)traction (i.e.. 
subtraction with a plus or minus sign) to provide bina- 



rized images, from which an area of variations for each 
image is calculated. Each area thus obtained is then 
surface integrated in terms of k>rightness to provide a 
signed rate of change for each changed portion. In step 
S4. the signed rate of change is processed in a serial 
manner each time tiie CCD camera 2 takes one frame 
to produce a result on which respiration counting is car- 
ried out in step S5. 

[0006] In this manner, the conventional respiration 
monitoring system can monitor the state of respiration 
through visual sensing in a non-restrictive and continu- 
ous manner. 

[0007] With this prior system, however, a supervisor 
who also acts as an operator has to input manually a 
specific region of interest (ROI) setting among tiie pic- 
tijres or images photographed by the CCD camera 2. for 
which region image processing is to be performed for 
measuring or counting the number of respirati(His per 
unit time, while watching the monitor 4. referred to 
above. 

[0008] Thus, with the conventional system in which 
tiie ROI setting is input manually, if, for instance, a 
patient to be monitored or photographed has moved to 
displace the ROI. it is difficult to measure the numk>er of 
respirations per unit time in an accurate manner. In 
practice respiration measurement becomes impossible 
when such a situation is left unattended. As a result, it is 
impossible to measure automatically the respiration rate 
for a long period of time. On the other hand, if the devi- 
ation or displacement of the ROI (i.e., movement of the 
person) is monitored by a supervisor to allow long-term 
accurate respiration counting, personnel expense and 
tiie amount of latxiur cannot be reduced. 
[0009] Moreover, in tiiis case, a change in the value of 
surface integration becomes small depending upon the 
ROI setting, so the accuracy of respiration counting can 
be reduced due to noise or other interference, or 
becomes difficult to recognise inhalatfon and exhalation 
of the person on a monitor. 

[0010] Furthermore, witii the conventional respiration 
monitoring system, the frame interval a for subtraction 
is also set and input by the operator, so similarly to the 
setting of tiie ROI. this gives rise to difficulty in long-term 
respiration counting. The frame Interval a is input by tiie 
operator based on his or her experience, so the accu- 
racy of respiration counting depends on the setting of a 
and sometimes decreases due to an inappropriate set- 
ting, with ttie result that recognition of inhalation and 
exhalation on ttie monitor becomes difficult. 
[0011] Further, the conventfonal respiration monitor- 
ing system calculates the rate of change in the darkness 
or grey level within the ROI using an inter-image sub- 
traction witii a plus or minus sign, and hence, if an 
increase and a deaease of brightness within the ROI 
are equal to each other in tiieir areas, it is determined 
that ttiere is no change in the rata 
[001 2] According to a first aspect of the present inven- 
tion, there is provided region-of-interest setting appara- 
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tus for respiration monitoring comprising: 

imaging means (such as a camera) for imaging an 
object to be monitored; 

5 

provisional-region setting means for extracting 
changed portions between successive images 
taken by the imaging means and setting an optimal 
one of the thus extracted changed portions as a 
provisional region; io 

determining means (e.g. an optimal-ROI determin- 
ing stage) for determining whether the provisional 
region set by the provisional-region setting means 
can be adopted as a changed region for respiration is 
counting; and 

region-of-interest setting means (e.g. an ROI set- 
ting and storing stage) for setting the provisional 
region as a region of interest for respiration count- 20 
ing when the determining means determines that 
the provisional region can be adopted as a changed 
region for respiration counting. 

With the above arrangement, it is possible to set the 25 
region of interest in an automatic manner. 
[0013] In a preferred embodiment, the provisional- 
region setting means comprises: 

subtracted-image storing means (e.g.. an original 3o 
image memory and a subtraction and summation 
circuit) for calculating absolute values of subtrac- 
tions between every successive two of a plurality of 
frame images taken during at least half a respira- 
tion period, and for summing and storing the abso- 35 
lute values of subtractions of the images thus 
obtained for predetermined frames; and 

region-of-interest candidate setting means for cal- 
culating the position and size of each of the 40 
changed regions based on brightness information 
of each subtracted image stored in the subtracted- 
image storing means and setting at least the great- 
est among the changed regions as a provisional 
region. 45 

[001 4] With the above arrangement, for a candidate of 
the region of interest, a single region can be selected in 
which there can be seen changes over the largest area, 
and hence it is possible to select as an ROI candidate a so 
region for which a sufficient respiration measuring or 
counting accuracy can be ensured in the presence of 
noise. 

[0015] In another preferred embodiment, the region- 
of-interest candidate setting means (e.g.. a candidate- ss 
region extracting stage) Is arranged to calculate the 
position and size of each of the changed regions based 
on brightness (change) information of each subtracted 



image stored In the subtracted-image storing means 
and to select an optimal one of the changed regions 
successively from the greatest to the smallest thereof 
as a provisional region. 

[001 6] With the above arrangement, in cases where a 
large region is not used for setting a region of interest, 
the next candidate can be retrieved, thus obviating the 
necessity of again performing a provisional region set- 
ting operation. This leads to an enhancement in 
processing. 

[0017] In a forther preferred embodiment the deter- 
mining means determines whether in the provisional 
region set by the provisional-region setting means, a 
grey level histogram indicative of distribution of the 
number of pixels for respective grey level values 
includes a twin peak characteristic having twin peaks 
each equal to or higher than a predetermined value. In 
one embodiment, the determining means comprises an 
optimal-ROI determining stage, and the predetermined 
value is set In such a manner that peaks of the twin- 
peak characteristic can be detected in the presence of 
noise. 

[0018] In this way, on the basis of the grey level histo- 
gram having a vertical difference (i.e., difference 
between the highest and lowest points) not less than a 
predetermined value, it is possible to select, among the 
ROI candidates, an optimal region of interest in which a 
sufficient difference in contrast can be detected to pro- 
vide a satisfactory accuracy in optical measurements. In 
this case, since the twin peak characteristic is a 
required condition, reliability in maintaining measuring 
accuracy can be further enhanced. 
[001 9] In a yet further prefenred form the determining 
means (an optimal-ROI determining stage) determines 
whether the area of tiie changed region is equal to or 
greater than a predetermined value. This allows not 
only selection of an optimal region of interest for which 
sufficient measuring accuracy can be obtained in the 
presence of noise, but also easily to recognise varia- 
tions in exhalation and inhalation of the object such as a 
patient on a monitor for example. 
[0020] In a still further prefenred form of, when the 
determining means (an optimal-ROI determining stage) 
determines that the provisfonal region set by the provi- 
sional region setting means cannot be adopted as a 
changed region for respiration counting, the provisional 
region setting means again performs the provisional 
region setting. Accordingly, the process of setting a 
region of interest is carried out repeatedly until an opti- 
mal region of interest is set, thereby increasing reliability 
in ROI setting. 

[0021] In a further prefened form, the apparatus fur- 
tfier comprises region-of-interest setting starting-condi- 
tion determining means (e.g., an object detecting stage) 
for detecting, as a condition for starting the setting of a 
region of interest, whether the object to be monitored 
has come into a field of view of the imaging means. 
[0022] With this arrangement, even if the patient has 
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moved away from the bed. when the patient returns, a 
region of interest is automaticaliy set once again, so 
respiration rTK)nitoring can be resumed in an automatic 
manner. 

[0023] According to a second aspect of the invention, s 
there is provided a respiration monitoring system com- 
prising: 

imaging means for imaging an object to be moni- 
tored; 10 
setting means for setting a region of interest for res- 
piration counting in images taken by a imaging 
means; 

change detecting means (e.g.. an ROI image 
extracting stage, a previous-frame memory, a is 
frame-memory selecting stage, an inter-image sub- 
traction stage and a surface integration stage 25) 
for calculating absolute values of subtractions 
between successive images produced by the imag- 
ing means in the region of interest set by the setting 20 
means and for summing up the absolute values of 
subtractions for every predetennined number of 
frames; 

counting means (e.g.. a serial data storing stage 
and a respiration calculating stage) for counting the 25 

number of respirations per unit time based on a 
trend of time-series changes obtained by the 
change detecting means. 

[0024] With the above arrangement, the atisolute val- so 
ues of sukrtractions between successive images are 
summed for every predetermined nurrtber of frames, so 
that even if the brightness in the region of interest 
changes with the area of increasing brightness equal to 
the area of decreasing brightness, such changes can be 35 
detected correctly, thus preventing errors in measure- 
ments which would othenAfise result from failure to 
detect the changes as in the prior art. 
[0025] In a preferred form, the system further com- 
prises condition determining means for determining. 40 
based on a frequency analysis of a trend of time-series 
data, whether the state of respiration is normal 
[0026] With this arrangement, discrimination can be 
clearly made as to whether the trend of time-series data 
is generated based on noise and the like, or on normal 4S 
respiration. 

[0027] In another preferred form, the system further 
comprises existence determining means for determin- 
ing whether the object to be monitored is outside a pre- 
determined imaging range. In tiiis way it can be so 
determined whether, for example, a patient to be moni- 
tored has moved out of a range to be monitored. This 
may happen when, for example, the patient gets out of 
bed for lingering or some other reason, or falls down 
from the bed. This serves to improve reliability in respi- ss 
ration monitoring. 

[0028] In a further preferred form, the respiration 
counting means counts tiie number of respirations per 
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unit time by measuring the nurrdser of peaks in a change 
of a trend in the time series data, and wherein a thresh- 
old is used which is variably set for each peak for count- 
ing thereof. Accordingly, the number of respirations per 
unit time can be reliably counted. That is. the magnitude 
of each peak occurring in the trend of the time-series 
data gen^lly changes, and in such a case, if the 
number of peaks is counted with a constant threshold, 
there will be a problem that p^ks smaller than the 
threshold cannot be counted. To counter this, the 
threshold is set according to tiie magnitude of each 
peaK so that the number of peaks can be counted cor- 
rectiy irrespective of variations in the peak magnitude. 
[0029] In a still further prefened form, the system fur- 
ther comprises frame-interval setting means (typically a 
frame-interval setting stage) for setting the predeter- 
mined number based on the summed value calculated 
by the change detecting means or tiie number of respi- 
rations per unit time counted by the counting means in 
such a manner as to emphasize changes in grey levels. 
With this arrangement, measuring accuracy can be 
inproved, and obsen/ation of changes on a monitor for 
example can be effected witii ease. 
[0030] In a yet further preferred form, the means for 
setting a region of interest for respiration counting in 
images taken by the imaging means comprises the 
region-of-interest setting means (an ROI setting stage) 
of the region-of-interest setting apparatus for respiration 
nK>nitoring as set out above. With the above arrange- 
ment, the entire process of respiration monitoring 
including the setting of a region of interest can be auto- 
mated so that respiration of the patient can be moni- 
tored for an extended period of time without requiring 
any human monitoring.. 

[0031] In a further preferred form, the provisional 
region setting means and/or the determining means 
operate during respiration counting. Therefore, in cases 
where the ROI needs to be set again during respiration 
counting, such new setting can be readily made imme- 
diately, thus avoiding intenuption of respiration monitor- 
ing for a long period of time. 

[0032] The invention will now be described by way of 
example with reference to the drawings in which:- 

Rg. 1 is a block diagram showing the overall 
arrangement of a first respiration monitoring system 
in accordance with the invention; 

Rg. 2 is a flow chart showing the overall operation 
of the respiration monitoring system; 

Rg. 3 is a blod^ diagram showing details of an ROI 
setting stage and a respiration measuring stage. 

Rg. 4 is a flow chart showing the operation of ttie 
ROI setting stage; 

Rg. 5 is a representation of image change informa- 
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tion obtained by a subtiacting and summing drcuit 
of the system: 

Rg. 6 is a grey level histogram; 

Rg. 7 is a flow chart showing respiration counting; 

Rg. 8 is a time chart showing the respiration speed 
or rate in terms of the absolute value of subtraction; 

Rg. 9 is a block diagram showing an optimal-ROt 
determining stage of a second respiration monitor- 
ing system in accordance with the present inven- 
tion; 

Rg. 10 is a block diagram showing a respiration 
rate calculating stage of a third respiration monitor- 
ing system in accordance with of the invention: 

Rg. 1 1 is a block diagram showing an exemplary 
method for frequency analysing time-series data in 
a fourth respiration monitoring system in accord- 
ance with the invention; 

Rg. 12 is a block diagram showing the overall 
arrangement of a fifth respiration monitoring system 
In accorclance with the invention: 

Fig. 13 is a block diagram showing a conventional 
respiration monitoring system; and 

Rg. 14 is a flow chart showing the operation of the 
conventional respiration monitoring system. 

EMBODIMENT 1 

[0033] Referring to Rg. 1. a respiration monitoring 
system comprises a CCD camera 2 for sensing or imag- 
ing the chest or thereabouts of a target object such as 
patient, who is being cared for, a region-of-interest 
(ROI) setting stage 10 connected to an output of the 
CCD camera 2 for setting a region of interest a respira- 
tion counting stage 20 connected to an output of the 
ROI setting stage 10 and the output of the CCD camera 
2 for counting the number of respirations per unit time in 
the region of interest (ROI) which Is set by the ROI set- 
ting stage 10. an abnormality determining stage 30 con- 
nected to an output of the respiration counting section 
20 for determining whether the number of respirations 
per unit time measured is normal or abnormal, and an 
object detecting stage 40 oonnected to an output of the 
abnormality determining stage 30 and tiie output of the 
CCD camera 2 for detecting the target object to be mon- 
itored. 

[0034] The object detecting stage 40 detects the tar- 
get object coming into the view in the CCD camera 2 
upon setting of the ROI by continuously detecting a 
large displacement of the target object on a picture 



image for a predetermined p^od of time, and gener- 
ates a detection or output signal wtvch acts as a trigger 
to actuate the ROI setting stage 10. Also, when the tar- 
get object tenporarily moves away from a bed or tiie 

5 like, the abnormality determining stage 30 determines 
an abnormality in the numba' of resFMraUons per unit 
time measured, and generates a corresponding output 
signal, based on which the object detecting stage 40 is 
started to prepare for watching the bed or thereabouts 

w and again detects the target object when it has returned 
to the bed, thus reactuating the ROI setting stage 10. 
[0035] Referring to Rg. 2. in step S1 1 . the system is 
actuated, and in step SI 2. the object detecting stage 40 
starts to detect the presence and absence of a target 

15 object, i.e., whether a target object is on the bed. Upon 
detection of tiie target object, the ROI setting stage 10 
sets a region of interest (ROI) in step SI 3. Upon setting 
of the ROI ttie ROI setting stage 10 sets a candidate for 
the ROI and determines the validity or feasibility thereof. 

20 If it is determined tiiat the setting of the ROI candidate is 
Invalid, the control process or flow returns to ttie exist- 
ence detecting step S12. and the above operations in 
steps 812 and S13 are repeatedly carried out until a 
valid ROI candidate is obtained becomes valkJ. 

25 [0036] If the ROI candidate setting is determined to be 
valid in step 813. the flow then goes to step 814 where 
respiration counting is carried out. Subsequentiy, in step 
815, if it Is determined that tiie number of respirations 
per unit time or the movement of the target object (e.g., 

30 body movement of a person to be monitored) Is abnor- 
mal, there may be the possibility that the target object 
be away from the bed. and thus, the control process 
returns to the existence detecting step 812. whereas if 
tiie determination In step 815 is otiienMse (i e., abnor- 

35 mal). an alarm is generated by the abnormality deter- 
mining stage 30. 

[0037] Rg. 3 shows In block form the details of the ROI 
setting stage 1 0 and the respiration counting stage 20 of 
the respiration monitoring system. 

40 [0038] The ROI setting stage 1 0 comprises an original 
Image memory 1 1 connected to an output of a preproc- 
essing circuit 6 that performs waveform shaping and the 
like, for continuously taking in the image obtained by the 
CCD camera 2 for a period of time just half the respira- 

45 tion period. A subtracting and summing circuit 1 2 is con- 
nected to an output of the original Image memory 1 1 for 
extracting a changed portion of the image stored 
tiierein, and a candidate-region extracting stage 13 is 
oonnected to an output of tiie subtracting and summing 

50 circuit 12 for labelling tiie summed image obtained by 
the subtracting and summing circuit 1 2 to provide a can- 
didate region for tiie ROI. An optin^l-ROI determining 
stage 14 is connected to an output of the candidate- 
region extracting stage 13 and the output of the original 

55 image menwry 11 for determining whether tiie region 
(image region) extracted by the candidate-region 
extracting stage 13 can be used as the ROI and an ROI 
setting and storing stage 1 5 is connected to an output of 
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the optimat-ROt determining stage 14 for setting as the 
ROl the Image region determined to be appropriate or 
optimal for the ROl by means of the optimaJ*ROI deter- 
mining stage 1 4. 

[0039] Here, it is to be noted that an operation control- 
ler (not shown) is connected to an output of the object 
detecting stage 40 shown in Fig. 1 . 
[0040] The respiration counting stage 20 comprises 
an ROl image extracting stage 21 connected to the out- 
put of the preprocessing circuit 6 and the output of the 
ROl setting and storing stage 15 for extracting from the 
image data obtained by the CCD camera 2 only those 
image data which lie within the ROl and a previous- 
frame memory 22 connected to an output of the ROl 
image extracting stage 21 for storing ROl image data for 
n frame images. An inter-image subtraction stage 24 is 
connected to an output of the ROl image extracting 
stage 21 and an output of a frame m^ory selecting 
stage 23 (described later) for performing subtraction 
between images spaced from each other a distance of 
a frames. A surface integrates stage 25 connected to an 
output of the inter-image subtraction stage 24 for sur- 
face integrating the values of the inter-image subtrac- 
tions thereby obtained. 

[0041 ] The respiration counting stage 20 comprises a 
time-series-data storing stage 26 connected to an out- 
put of the surface integration stage 25 for storing, as 
serial data, the values of surface integration obtained by 
surface integrating the inter-image subtraction in a time 
serial manner for a predetermined period of time, and a 
respiration rate calculating stage 27 for calculating the 
number of respirations per unit time based on the data 
stored in the time-series-data storing stage 26. A frame 
inten^al setting stage 28 is connected to an output of the 
respiration rate calculating stage 27 for setting a frame 
interval a with which the inter-image subtraction stage 
24 performs inter-image subtraction, based on the 
result of calculation by means of the respiration rate cal- 
culating stage 27, and the frame memory selecting 
stage 23 is connected to an output of the frame interval 
setting stage 28 for selecting from the previous-frame 
memory 22 a previous frame image spaced an interval 
of a from a current image and outputting it to the inter- 
image subtraction stage 24. 

[0042] Now. the operation of ROl setting stage 10 will 
be described with reference to the flow chart of Fig. 4. 
[0043] First, in step S21 , images from the CCD cam- 
era 2 are stored into the original image memory 1 1 for a 
period of time equal to half the respiration period. In 
step S22, the subtracting and sunvning circuit 12 per- 
forms subtraction at a predetermined sut>traction inter- 
val (e.g., one frame), and sums up information of 
changes which are given by absolute values of subtrac- 
tions. An example of such changes of information thus 
summed is shown in Fig. 5. in which of dark portions 
indicate image portions for which changes have been 
recognised during a time equal to half the respiration 
period. In step S23. the candidate-region extracting 



stage 13 cames out labelling processing on the informa- 
tion of image changes depicted in Fig. 5, and calculates 
an area and infornriation of a rectangle enclosing the 
area for each label, thereby setting an ROl candidate. 

5 [0044] Although in this embodiment, a plurality of ROl 
candidates with labels having gradually decreasing 
areas from the largest to smallest have been prepared, 
a single candidate having a label of the largest area can 
instead be used, and if it is determined according to an 

10 optimal-ROI determination to be described later that 
this candidate is not c^mal. the control process returns 
to step S21 so as to set a new ROl candidate once 
again. 

[0045] (n st^ S24, tiie optimal-ROI determining stage 

15 14 determines an optimal ROl. Specifically, in this opti- 
mal ROl determination, an original image date related 
to the position and size of a rectangular region informa- 
tion set in the first candidate is obtained from the origi- 
nal image memory 11. thus providing a grey level 

20 histogram indicative of distritjution of the number of pix- 
els at each grey level. The values of grey levels are 
determined according to 255 steps or degrees of grada- 
tion. An example of such a grey level histogram is 
shown in Fig. 6. In the grey level histogram, it is deter- 

25 mined whether a twin peak characteristic including two 
adjacent peaks at a short interval can be recognised 
and whether the area of the region (i.e., rectangular 
region) listed as a candidate is equal to or greater than 
a predetermined value. If these conditions are not satis- 

30 fied. the second candidate is taken as the next candi- 
date for its optimality determination according to this 
embodiment, as referred to above. 
[0046] Subsequentiy, if the conditions of determina- 
tion are satisfied in step S24. tiie position and size or 

35 rectangular area of the optimal ROl are stored in the 
ROl setting and storing stage 15. and thus the optimal 
ROl Is finally set in step S25. On the other hand, if the 
conditions of determination are not satisfied, then the 
control process returns to step S21 . thus repeating the 

40 optimal ROl setting processing. 

[0047] After the optimal ROl has been set as 
described above, the respiration counting stage 20 or 
counts the number of respirations per unit time using 
the set ROl. 

45 [0048] The operation of the respiration counting stage 
20 will now be described with reference to Fig. 7. 
[0049] Rrst. In step S31, image data within tiie set 
ROl is fed from the CCD camera 2 to the previous-frame 
memory 22 and stored therein. Subsequentiy, in st^ 

so S32. the inter-image subtraction stage 24 performs 
inter-image subtraction so as to provide an absolute 
value of subtraction for each pixel, which is then surface 
integrated by tiie surface integration stage 25 in step 
S33. 

55 [0050] The above processes are carried out in a time- 
series manner, as shown in step 334. and the data 
obtained during such processings are stored in the 
time-series-data storing stage 26. based on which data 
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the respiration counting stage 20 counts the numt)er of 
respirations per unit time in step S35. 
[0051] Fig. 8 illustrates in real time the time-series 
data obtained by the surface integrating stage 25. In this 
diagram the respiration speed is shown by the absolute 
value of subtraction, and as will be described, there 
clearly appear a plurality of twin peaks each having a 
pair of peaks indicative of exhalation and inhalation, 
thus emphasising changes or variations despite noise 
interference.. 

[0052] Here, the number of respirations per unit time 
is measured by counting the number of peaks exceed- 
ing a constant or fixed threshold Sh 
[0053] In step S36. the frame-interval setting stage 28 
automatically sets a frame interval in the form of the 
interval of subtraction a based on the number of respi- 
rations per unit time as measured in such a manner as 
to ernphasise changes in the surface integration value 
(grey level), and outputs it to the frame memory select- 
ing stage 23 which then sheets, based on the sut)trac- 
tion interval a, a frame memory having a frame image 
stored therein previously apart an interval of a frames 
from the cunent image, and outputs the data therein to 
the inter-image subtraction stage 24. 
[0054] Here, it is to be noted that for the subtraction 
interval a, a frame interval corresponding for example, 
to half the respiration period is selected in order to 
emphasise changes in the grey level or darkness. In 
view of this, the number of frames stored in the previ- 
ous-frame memory 22 is selected according to the 
number of frames per unit time and the respiration 
period. The previous-frame memory 22. in which the 
data used for inter-image subtraction was stored, now 
becomes available for storing the next image data. In 
cases where the frame interval a is set according to the 
respiration period (or the nunr^er of respirations per unit 
time), as stated above, a change In the surface integra- 
tion obtained may be multiplied by l/o, in order to indi- 
cate the surface integration change at the same rate. 

EMBODIMENT 2 

[0055] In the aforementioned embodiment 1 . the res- 
piration counting stage 20 perforrr^ respiration counting 
using the ROI set by the ROI setting stage 10. but in this 
case, the ROI setting stage 10 may be constructed such 
that it is always preparing for setting a new ROI during 
respiration counting by the respiration counting stage 
20. In this case, as shown In Fig. 9. an optimal^ROI 
determining stage 14A is provided with a temporal stor- 
age stage 14a for storing an ROI each time the ROI is 
determined to be optimal by means of the optlmal-ROI 
determining stage 14A. Thus, for exanple. if the abnor- 
mality determining stage 30 determines the presence of 
abnormality, the ROI setting stage 10 can immediately 
set again the optimal ROI stored in the temporal storage 
stage 14a in place of the current ROI set in the ROI set- 
ting and storing stage 15. In tiiis manner, it is possible to 



avoki interrupting respiratk>n counting for the time 
required to freshly set a new ROI thereby enabling res- 
piration counting to restart immediately. If it is again 
determined that the number of respirations per unit time 
5 freshly counted is abnormal, the abnormal respiration is 
immediately decided to be not due to inappropriate set- 
ting of the ROI thus enabling speedy recovery treat- 
ments to be taken. 



[0056] Although in the aforementioned embodiment 1 . 
tiie respiration rate calculating stage 27 counts the 
number of peaks using the constant or fixed threshold 
Sh. as shown in Fig. 8, the threshold Sh may be varied 
for each peak. Fig. 1 0 shows in block form an exemplary 
arrangement of a respiration rate calculating stage 27A 
in such a case. The respiration rate calculating stage 
27A comprises a differential circuit 27a and a hold cir- 
cuit 27b t)oth connected to an output of a time-series 
data storing stage 26 (see Fig. 3), a subtracter 27c con- 
nected to outputs of the hold circuit 27b and the time- 
series data storing stage 26, a setter 27e in which a pre- 
determined value is set. a comparator 27d connected to 
outputs of the subtractor 27c and the setter 27c. and a 
counter 27f connected to an output of the comparator 
27d. 

[0057] With the above anangement, the differential 
circuit 27a differentiates the time-series data stored in 
the time-series data storing stage 26 to find its peak, 
which is then output to the hold circuit 27b and there 
held tiierein as a peak value. The subtractor 27c calcu- 
lates tiie difference between the peak value and the 
subsequent time-series data, and outputs tine calcu- 
lated difference value to the comparator 27d in which 
the difference value is compared witii the set value in 
the setter 27e. The comparator 27d generates an output 
to tiie counter 27f when the difference value Is greater 
tiian the set value. The counter 27f counts the number 
of outputs from the comparator 27d. thereby providing a 
count of peaks. 

[0058] In the respiration rate calculating stage 27A, if 
a peak value varies, tiie threshold therein is accordingly 
changed to detect such a varying peaK so that the 
number of peaks can be counted accurately irrespective 
of the magnitude of each peak. 

EMBODIMENT 4 

[0059] In embodiment 1 described above, the abnor- 
mality determining stage 30 is constructed such that it 
makes an abnormality determination based solely on 
the number of respirations per unit time as counted by 
the respiration rate calculating stage 27. However, there 
may sometimes be a case where the abnormality deter- 
mining stage 30 cannot measure the true respiration 
(i.e. respiratory movements) correctiy. since the ROI 
once property set departs from its original or intended 
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position due to movements of the object to be moni- 
tored, or the influence of noise affecting respiration 
counting. In view of this, to determine whether respira- 
tion is counted correctly, a frequency spectrum of a 
time-series data can t>e obtained for comparison with a 
frequency spectrum of a normally or correctly measured 
respiration. Such a comparison can be made using a 
statistical test of hypothesis. 

[0060] Rg. 1 1 shows in t)lock form an example of such 
an arrangement. In this figure, a fast Fourier transform 
(FFT) stage 51 is provided at an output side of the time- 
series data storing stage 26 for generating a frequency 
spectrum of the time-series data output. A comparator 
52 compares the frequency spectrum of the time-series 
data output from tiie FFT stage 51 with a normal or cor- 
rect frequency spectrum set in a frequency spectrum 
setting stage 53. This comparison is made, for example, 
in terms of the magnitude or strength of a predeter- 
mined frequency component. An abnormality determin- 
ing stage 30A determines, whetiier tiie respiration 
counting is normal based on the comparison of compa- 
rator 52. 

EMBODIIi/IENTS 

[0061] Although in the above-described embodiment 
1 , the object detecting stage 40 detects an object com- 
ing into or tine field of view of the CCD camera 2 upon 
setting an ROI by detecting a large change in time of the 
images produced by the camera 2, it may also detect 
the object going out of the field of view, again by detect- 
ing a large change in time of the images. A large change 
in time of the images can be detected, for example, by 
detecting a change in contrast or brightness of the 
images at predetermined time inten/als by way of sens- 
ing conrelation between the images or a change in the 
grey level histogram thereof. In this enfd)Odiment. then, 
an Object detecting stage 40A may be provided with an 
image memory for tiie above purpose, but instead the 
original image memory 11 of the ROI setting stage 10 
can be used for investigating correlation between the 
images, or grey level histograms can be compared with 
each other at a predetermined time interval using a sta- 
tistical test of hypothesis so as to perform the above 
detection. Detection of the object to be monitored hav- 
ing gone out of the field of view is signalled to an abnor- 
mality determining stage 30B. so that it is possible to 
deal with the abnormal condition while discriminating 
respiratory abnormality from such an abnormal situa- 
tion in which normal respiration counting can not be 
effected due to the absence of an object to be watched 
in the field of view of the camera 2. 
[0062] As described above, automatic setting of the 
ROI can provide the advantage of automatic accurate 
respiration counting for an extended period of time with- 
out requiring operator monitoring. In addition, exhala- 
tion and inhalation can quite easily be recognised on a 
monitor. 



Claims 

1 . A region-of-interest setting apparatus for respiration 
monitoring conprising: 

5 

imaging means for imaging an object to be 
monitored; 

provisional-region setting means for extracting 
changed portions between successive images 
10 produced by the imaging means and selecting 

one of tiius extracted changed portions as a 
provisional region; 

determining means for determining whether 
the provisional region set by the provisional- 
's region setting means can be adopted as a 
changed region for respiration counting; and 
region-of-interest setting means for setting the 
provisional region as a region of interest for 
respiration counting when the determining 
20 means determines that the provisional region 
can be adopted as a changed region for respi- 
ration counting. 

2. Apparatus according to claim 1 . wherein the provi- 
25 sional-region setting means comprises: 

subtracted-image storing means for calculating 
absolute values of subti-actions between every 
successive two of a plurality of frame images 
30 taken during at least half a respiration period. 

and for summing and storing the absolute val- 
ues of subtractions of the images tiius obtained 
for every predetermined frame; and 

35 region-of-interest candidate setting means for 

calculating the position and size of each of the 
changed regions based on brightness informa- 
tion of each subtracted image stored in the 
subb'acted-image storing means and setting at 

40 least tiie greatest among tiie changed regions 

as a provisional region. 

3. Apparatus according to claim 1 . wherein the provi- 
sional-region setting means comprises: 

45 

subtracted-image storing means for calculating 
absolute values of subtractions between every 
successive two of a plurality of frame images 
taken during at least half a respiration period. 

50 and for summing and storing the absolute val- 

ues of subtractions of the images tiius obtained 
for every predetermined frame; and 
region-of-interest candidate setting means for 
calculating the position and size of each of the 

55 changed regions based on brightness informa- 

tion of each subtracted image stored in the 
subtracted-image storing means and setting an 
optimal one of the changed regions succes- 
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sively from the greatest to the smallest thereof 
as a provisional region. 

4. Apparatus according to any of claims 1 to 3, 
wherein the determining means is arranged to s 
determine whether, in the provisional region set by 
the provisional-region setting means, a grey level 
histogram indicative of distribution of the number of 
pixels for respective grey level values includes a 
twin peak characteristic having twin peaks each io 
equal to or higher than a predetermined value. 

5. Apparatus according to claim 4. wherein the deter- 
mining means is ananged to determine whether the 
area of the changed region is equal to or greater is 
than a predetermined valua 

6. Apparatus according to any of claims 1 to 5. 
wherein, when the determining means is arranged 

to determine that the provisional region set by the 2o 
provisional region setting means cannot be 
adopted as a changed region for respiration count- 
ing, the provisional region setting means is 
an-anged to perform again the setting of a provi- 
sional region. 25 

7. Apparatus according to any of claims 1 to 6, further 
comprising region-of-interest setting starting-condi- 
tion determining means for detecting, as a condition 

for starting the setting of a region of interest. 30 
whether the object to be monitored has come into a 
field of view of the imaging means. 

8. A respiration monitoring system comprising: 

35 

Imaging means for imaging an object to be 
monitored: 

setting means for setting a region of interest for 
respiration counting in images produced by the 40 
imaging means: 

change detecting means for calculating abso- 
lute values of subtractions between successive 
images produced by the imaging means in the 45 
region of interest set by the setting means and 
for summing tiie absolute values of subtrac- 
tions for every predetermined number of 
frames: and 

so 

counting means for counting the number of res- 
pirations per unit time based on a trend of time- 
series changes obtained by the change detect- 
ing means. 

55 

9. A system according to claim 8, further comprising 
condition determining means for determining, 
based on a frequency analysis of a trend of time- 



series data, whether the state of respiratton is nor- 
mal. 

10. A system according to claim 8 or claim 9. further 
comprising existence determining means for deter- 
mining whether the object to be monitored has 
moved out of a predetermined imaging range. 

11. A system according to any of claims 8 to 10, 
wherein the respiration counting means is arranged 
to count the number of respirations per unit time by 
measuring the numk>er of peaks in a change of a 
trend in the time series data, and wherein a thresh- 
old is used which Is set for each peak for counting 
thereof. 

1 2. A system according to any of claims 8 to 1 1 . further 
comprising frame-interval setting means for setting 
the predetermined number based on the summed 
value calculated by the change detecting means or 
the number of respirations per unit time counted by 
the counting means in such a manner as to empha- 
sise changes in grey levels. 

13. A system according to any of claims 8 to 12. 
wherein the means for setting a region of interest 
for respiration counting in images taken by the 
imaging means comprises the region-of-interest 
setting means of the region-of-interest setting 
apparatus for respiration monitoring as defined in 
any of claims 1 to 7. 

14. A system according to daim 13. wherein the provi- 
sional region setting means and/or the determining 
means operate during respiration counting. 



45 



so 



9 



EP 0919 184 A1 



CO- 



^ 

Respiration 
counfing 
stage 



ROI setting 
stage 



Object 
detecting stoge 



'10 



Count 



30 



'I 

Abnormality 
determining stage 



Fig.1 



Existence detecting 
step 



ROI setting 
step 



Respiration 
counting step 



Fig.2 



Normal 



Overall 
operation 



) 



1 




Actuate 
system 



r 



S12 



Detect 
existence 




S13 
Invalid 



Volid/ 



Count 
respirotion 



/S15 



Check body movements' 
end abnormality in 
respiration 

[Abnormal 



10 



EP 0919 184 A1 



t . 



C7^ 



o 

c 
o 



Q. 

a; 



CD 
CM 



c 
o 

Q-QtC 01 



m 









> 






















E 




a 




Fr 


or 
1/1 



CNI 



E 



1 



10 



01 

C H oi 

' q; o CT» 

01 a VI 



I 



c 


01 


0 


o> 




a 


Ol 


C/> ' — - 


u 


ing 

liny 


a 






w 01 








£ jg 






cr 


X — 




a; 



a. i/» 



01 

c 

0/ ° a; 



E 3 " 
— 00 (/I 



1 



c3 a; 
E y 

O ^ i/i 







cn 




c 








— 


rt 




"L- E Q/ 


XI f 




3 : 




00 » 


/I </» 




_ J 




Of 








"3 




> 








o; 




c 




E 




-2? 




Oi 






"t— 




Of 


OL 


w 












a 


ove 


JC 






a 


a 




O) 




ex. 




c 


c_ 


*-» 


< 1— 



cn 

a. 



11 



EP0919184A1 



(rOI semng ) 









Sfore images in memory 

for half a period 










Sum,absolufe values of 
subfrocfion befween images 
af interval of 7 frame 


/~S22 




r 




Calculafe area for each label 
and rectangle information 
enclosing the area and set 
ROI candidate 


^S23 








Determine whether grey level 
histogram has fwin peak 
characferisfic and whefher 
area is above a predetermined 
value 










Set ROI candidate if conditions 
of determination are satisfied, 
or return to step S21 if 
conditions of deferminafion 
are not satisfied . 


^S2S 



Fig. 4. 

12 



EP 0919 184 A1 





255 Value of grey level 



Grey level histogram 



Fig.6. 



13 



EP 0919 184 A1 



(Respiration ^ 
counfing J 
T 

Input image data -^^531 




Subtraction interval 
KoL : frames ) 



Calculate absolute values of 
subfraction between images 

I 



Surface integration in ROI 










Time-series 
processing 





532 



'S33 



•S34 



Count' the number of respiniKons 
per unit fime 



VS35 



^S36 



Fig.7. 



I 



14 



EP 0 919 184 A1 




EP 0 919 184 A1 



(V 

o.o ^ 

5 F 



CSI 




C7^ 




o 


1 

c 




-o 






Qiffere 
iqtion 
circuii 




o 

-JO 


circu 







q; 




or 




i/i 




1 


c 


Of 




E 


o 






E 


a 


o 






a 


u. 





16 



EP 0 919 184 A1 



11 



CM 

to 




to 




1 



Abnormality 
determining 
stage 




O 




Frequency 
spectrum 
setting stage 




Compar 





C71 



in 



CM' 



17 



EP 0919 184 A1 




CD 
CNI 





a; 


on 








'a 




"q. 


_c 




c 


Of 




a: 


O 







A. 



cn 




c 






or 






ai 


a 


1/1 








o 









c 



cu 
a; 



XI 

o 



Csl 



/ \ 



18 



EP 0919 184 A1 




19 



EP 0 919 184 A1 



(Respiration ^ 
counting J 



Input image data 



Subtraction interval 
(QC: frames) 



Signed subtraction 
between images 



•S2 



Surface integration 
in ROI 



S3 



Time-series 
processing 



S4 



Count number of 
respirations per 
unit time 



^S5 



Fig.K 



20 



EP 0 919 184 A1 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



AppllciUon Number 

EP 98 30 9569 



DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Citation of document with indicatioa where appropriate. 
of relevant passages 



Relevant 
to claim 



CLASSIFICATION OF THE 
APPLICATION (tnt.Ct.6} 



NAKAJIMA ET AL.: "Evaluation of body 
motion by optical flow analysis" 
JAPANESE JOURNAL OF APPLIED PHYSICS, 
vol. 36, no. 5a, May 1997, pages 
2929-2937, XP000732415 
JP 

* page 2929, right-hand column, line 22 - 
page 2933, right-hand column, line 7 * 

* page 2934, right-hand column, line 42 - 
page 2935, right-hand column, line 2 * 

* figures 1-5 ♦ 

JP 08 257015 A (ISHIKAWA ET AL.) 
8 October 1996 

* the whole document ♦ 

& US 5 704 367 A (ISHIKAWA ET AL.) 

* the whole document * 

DUFFY ET AL.: "Optical non-contact 
measurement of abdominal volume change 
during anaesthesia" 

lEE COLLOQUIUM ON OPTICAL TECHNIQUES AND 
BIOMEDICAL APPLICATIONS, 14 June 1991, 
pages 8/1-8/4, XP002095965 
London, GB 

* the whole document * 

US 4 975 960 A (PETAJAN) 4 December 1990 

* column 3, line 46 - column 5, line 14 * 

* column 6, line 6 - column 7, line 10 * 

* figures 1-3 * 



1,8 



A61B5/08 
A61B5/113 



1.8 
1,8 
1,8 



TECHNICAL FIELDS 
SEARCHED (lm.a.6) 



A61B 
G07C 



1,8 



The present search report has been drawn up for all claims 



THE HAGUE 



Dftts of ootTvtotion ot th« SMflCh 

9 March 1999 



Chen, A 



CATEGORY OF CITED DOCUMENTS 

X : particularly ralavant if takan alone 

Y : particulariy relevant V combined with another 

document of the eame category 
A : technological background 
O : non-wrilten disclosure 
P : intermediate document 



T : theory or principle underlying the Invention 
E : eariier patent document. bU pUiished on, or 

after the filing date 
D : document dted In the appUcatien 
L : document dted for other reasons 



A : member of the eame potent famtty. eorresponding 



21 



EP0d19184A1 



ANNEX TO THE EUROPEAN SEARCH REPORT 
ON EUROPEAN PATENT APPLICATION NO. 



EP 98 30 9569 



This annex lists the patent family members relating to the patent documents cited in the above-mentioned European search report. 
The members are as contained in the European Patent Office EDP file on 

The European Patent Office is in no way Gable fbr these particulars which are merely given for the purpose of information. 

09-03-1999 



Patent document 
cited in search report 



Publication 
date 



Patent family 
mQmber{s) 



Publication 

date 



JP 8257015 



08-10-1996 



US 4975960 



04-12-1990 



US 5704367 A 
NONE 



06-01-1998 



For more details about this annex : see Official Journal of the European Patent Office, iMo. 12/82 



22 



